What accuracy statistics really measure

This paper aims to provide the software estimation research community with a better understanding of the meaning of, and relationship between, two statistics that are often used to assess the accuracy of predictive models: the mean magnitude relative error, MMRE, and the number of predictions within 25% of the actuals, pred(25). We demonstrate that MMRE and pred(25) are, respectively, measures of the spread and the kurtosis of the variable z where z=estimate/actual. Thus, we consider z to be a measure of accuracy and statistics such as MMRE and pred(25) to be measures of properties of the distribution of z. We suggest that we need measures of the central location and skewness of z as well as measures of spread and kurtosis. Furthermore, since the distribution of z is non-Normal, we may need non-parametric measures of these properties. For this reason, boxplots of z are useful alternatives to simple summary metrics. We also note that the simple residuals are better behaved than the z variable and could also be used as the basis for comparing prediction systems.